Social Values at the Interface: Toward “Just” Human-Computer Ranking Designs at Scale

In the design of human-computer ranking systems for the adaptive display of information, designers often define a domain-specific scoring function which maps items such as people or information search results to numeric scores. Classic ranking systems typically display these items in a linear fashion, sorted by score. There are shortcomings to this approach: such ranking systems do not provide a diversity of results, and in aggregate the distribution of collective user attention is biased by the users’ trust in the quality of these orderings. Furthermore, ranking systems based on sorted orders embody a property of chaotic systems, namely that small perturbations in the input—the underlying scoring functions—may have large effects in the output—the distribution of collective user attention. Thus, we propose an alternative human-computer ranking system called donkey sort, which strikes a balance between complete order and uniform randomness, performing probability sampling of the display permutations found in a Latin square design.